Distinguishing Humans from Bots in Web Search Logs

Cleaning workload data and separating it into classes is a necessary pre-requisite for workload characterization. In particular, the workload on web search engines is derived from the activities of both human users and automated bots. It is important to distinguish between these two classes in order to reliably characterize human web search behavior, and to study the effects of bot activity. However, available workload data is not accompanied by labels that can be used as a basis for learning and generalization. To cope with the lack of labeled data, we suggest using two mechanisms. The first is to employ two thresholds for each criterion, enabling the identification of users who are most probably human or most probably bots according to need, and avoiding ambivalent cases. The second is the notion of “strong” criteria, which identify levels of activity which are highly unlikely or even impossible for humans to achieve. We then use an iterative process of refining the thresholds to combine the results of multiple metrics in a mutually consistent manner. Results using the AOL log identify over 92% of the users as human, and only a small fraction (0.6%) are probable bots. The humans tend to display relatively consistent behavior, whereas bots may exhibit markedly different behaviors. In particular, it is not uncommon for a bot to be very different from typical human behavior according to one criterion, while being indistinguishable from a human according to another.